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Introduction 



This is the first in a series of four modules about vocational 
competency measurement. The purpose of this module is to trace 
the growth of the formal measurement of skills, knowledge,, and atti- 
tudes in vocational programs, and to present, some uses of competency 
■testing in vocational programs. The next module, Module 18, discusses 
how to determine requirements for vocational compefency measures. 
Module 19 presents a step-by-step approach to developing the compe- 
tency tests, and Module. 20 considers approaches to validating com- 
petency tests and using test results. The discussion presented here 
is based on the experiences of the American Institutes for Research 
in conducting the Vocational Competency Measures (VCM) project for 
the U.S. Department of Education.. 



Overview • 

The first section of this module examines some of the early 
history and points out the importance of the military services in 
the development and use of competency measures. Current efforts 
are described in a review of ongoing projects that assess occupa- 
tional competencies of students in vocational training programs. 

In the second section, the components that make up a compre- „ 
hensive set of vocational competency measures are described. The 
discussion also covers and clarifies the definitions typically used 
for various testing techniques. 

The module concludes with an overview- of the ways vocational 
educators and administrators can use competency measures , to plan, 
evaluate, and improve vocational programs. Recommendations are 
included for using the components of a comprehensive competency test 
package to enhance communication among instructors, administrators, 
students, and parents. 

Instructions to the Learner '• » 

The Self-Check items and possible responses to them are found 
in the Appendices. These questions have two purposes. First, before 
you begin work on the module, you may use them to check quickly 
whether you have already learned the- information in previous classes . 
or readings. In some instances, with the consent of, your instructor, 
you might decide to skip a whole module or parts of one. The second 
purpose of the Self-Check is to help you review the content of modules 
you have studied in order to assess whether you have achieved the 
module's goals and objectives. 
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You can also use the list of goals and objectives that follows 
to determine whether the module content is new to you and requires 
in-depth study, or whether the module can serve as a, brief review 
before you continue to the next module. 



Goals and Objective's * 

Goal 1 : Summarize and explain the growth and significance of ^ 
student competency testing in vocational education programs. 

Objective 1.1 Describe the major contributions competency 
testing has made to vocational education programs. 

Objective 1.2 Identify some competency testing efforts that 
'historically have provided a foundation for current development. 

Objective 1.3 List three current major competency test 
^projects. ■ • . ^ < 

Goal 2 : Describe .the components of comprehensive student vocational 
competency measures, and define testing techniques making up those 
components . , ~ , 

Objective- 2.1 Identify the thre,e main instruments included, 
" in a comprehensive set of student competency measures. 

Objective 2.2 Define criterion- and norm-referenced testing. 

Goal 3 : Analyze the use of competency measures fo*r evaluation, 
instruction and administration, and, communication. 

Objective 3.'l Describe some uses of competency measures for 
.evaluating vocational programs^. 

* * * ■ 

Objective 3.2 Identify the major uses of * competency measures 
for improving, instruction and administration for vocational 
programs. 

i 3*3 Discuss the uses of comprehensive competer 

among 



Objective 3*3 Discuss the uses of comprehensive competency 
measurement for increasing and facilitating communication amc 



various interest groups. 
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Resources 



In order to complete the learning activities in this module, - 
you will need information contained in the following publication: 5 



Erickson, R,C. # , & Whittling, T.L. Measuring student growth: 
> ^Techniques and* procedures for occupational education 
' Urbana, 111.: Griffon Press, 1976* " 



O 



GOAL Is Summarize and explain the growth and significance of 
student competency testing in vocational education 



programs. 



Whv So Much Attention To Studen t Competency Testing! 



Traditionally the evaluation of vocational education has 
focused on learner outcomes, whether or not the vocational program 
was labeled objectives-based, performance-based, or competency- 
based. Most often the ."outcome" boiled down to the question: Did ^ 
the graduate obtain a job in a field directly related to the con- 
tent of the vocational training?" 

How can we argue with this ultimate of "bottom-line" indica- 
tors? What could possibly be wrong with so objective a measure as 
employment? For one thing, this presumes that the primary objective 
of all studehts is to obtain a job in the field where vocational 
training is received. Vocational educators know that this is not 
the case. Just ask any teacher of a high school auto shop course. ^ 

Another problem with the narrow focus on "employment as the 
criterion for evaluating vocational education, especially at the 
high school level, is that employment is heavily affected by both 
national and local economic conditions. In fact, the 1979 eyaiua- . 
tion report of the U.S. Department of Health, Education* and Welfare 
states that •'...economic conditions probably much more powerfully 
influence employment among youth than curriculum choice (p. 496). 

It is not surprising, then, that a growing number of individuals 
concerned with the evaluation of vocational education are also asking 
the question: How well can vocational program graduates do the jobs 
for which they were trained,? This increasing trend toward measuring 
student competencies (skills, knowledge, and attitudes) is reflected 
in a recent review of studies covering vocational education outcomes • 
(Taylor, Darcy, & Bollard, 1979). 

Current federal legislation supports this growth in competency t . 
measurement. For example, the Rules and Regulations for the voca- 
tional education section of the Educational Amendments of 197b 
(Public Law 94-482) prescribes that the State Board "evaluate in 
quantitative terms the effectiveness of each formally organized pro- 
gram or project supported by federal, state, and local funds. — 
terms of: 
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Results of student achievement as measured, for example, by: 

• St'andard occupational proficiency measures 

I 

• Criterion-referenced tests 
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• Other examination of students 1 skills, knowledge,, 
attitudes, and readiness for entering employment 
successfully. 11 

It 

But the evaluation of vocational program effectiveness is 
certainly not the only reason for the growing attention to the 
measurement of student competencies. If you are associated with 
a competency-based vocational education (CBVE) program, especially 
one that involves open entry or open exit, you know how critical 
student competency measurement is to many pr' gram functions — not 
just a program evaluation. 

Just look at the following features of exemplary CBVE programs 
that Russell found in a recent survey (1978, pp. 5-56) to see how 
closely CBVE is tied to student competency testing: 

• Pre-tes.ting students upon entry to determine the skills- 
. they already have as well as objectives that need to be 

achieved 

/ 

• Allowing each student to proceed to subsequent instruc- 
tion as soon as performance objectives are attained 

• Providing an alternative method of instruction if a , 
student does not achieve a learning task 

• Recording student performance as each objective is 
achieved 

• Placing greater emphasis on exit requirements (profi- 
ciency) than on entrance requirements 

• Assessing students on the basis of competencies 

Indeed, a comprehensive vocational competency measurement program 
can be a tremendous resource for teaching and administration from 
the time of initial assignment through graduation. It can also 
help satisfy the demands for information that students, parents, 
and employers are making upon vocational educators today. 

Even with all of these uses we have not exhausted the potential 
contribution that competency measures can make to the improvement of 
vocational education. In a joint report, the American Association of 
Community and Junior Colleges and the American Vocational Association 
noted the problem of articulation among vocational education delivery 
agencies and recommended the following for improving cooperation: 



The U.S. Office of Education should, through the Bureau of 
Occupational and Adult Education, develop a data bank of 
competencies needed by individuals to enter or qualify for 
work in a broad range of occupations. Appropriate criteria 
for. assessing whether or not the competencies are achieved 
should also be required (1978, pp. 22-23). 




Some Early History 

The formal measurement -of competency in task performance can 
trace its roots back to the early history of the testing movement. 
As noted by Hale (1982) in his comprehensive review of the history 
of employment testing, as far back as 1814 the Army had Jjl«ady 
.instituted examinations for surgeons. The military, in both World 
Wars I and II, contributed significantly to the use of tests tor 
personnel classification and performance evaluation. Chapman, in 
1921, noted that one of the important outgrowths of World War I Army 
Personnel Research was the develor-snt of the trade test. This 
instrument was devised, in his words, "to make it possible for a 
trained examiner, unskilled in any particular trade, to measure in 
objective terms the trade standing of any recruit claiming skill in ) 
any of the several hundred trades necessary to the work of the Army 
(p. v). 

Chapman, a member of the Army Trade Test Division of the 
Committee on Classification of Personnel, defined "trade" much the 
way "occupation" is defined .today, and hence the term. "trade test 
was used synonymously with "occupational test" and ^professional 
test."' It encompassed such diverse occupations as those of a sur- 
veyor-rcook, statistician, and typist. Trade ability, according to 
Chapman (1921,- p. 12)., signified "what is commonly meant 'by a ^ 
person's competency to follow a trade, occupation or profession. 

World War II and the decade that followed gave another boost 
to competency testing, with special emphasis on measuring the profi- 
ciency of aircraft pilots and equipment maintenance personnel 
(Flanagan, 1948). In an extensive review of this research and its 
applications, Glaser and Klaus (1962) discuss the problems of measur- 
ing proficiency of the human component in man-machine systems. 

Today, the military still stands as one of the major developers 
and users of occupational competency measures. Ot:her organizations, 
particularly large businesses, are also showing increasing interest 
in testing. While the number of these organizations is growing, much 
remains to be done before we can truly say that occupational compe- 
tency testing has reached its full potential.' In his sweeping 
criticism of the wholesale use of intelligence tests by schools, 
colleges, and employers, McClelland (1973) calls for competency test- 
ing as an alternative approach. As viewed by McClelland (p. 7), n the 
best testing is criterion sampling... There is ample evidence that 
tests which sample job skills 'will predict proficiency on the job... 
"Criterion sampling means that testers have got to get out of their 
offices where they play endless word and paper-and-pehcil games and 
into the field where they actually analyze performance into its 
components . 11 

In speaking about the trade testing movement in 1921, Chapman 
(p. vi) noted that the "movement is only in its infancy, but the 
methods that have been evolved will prove a firm foundation upon 
which an elaborate superstructure Can safely be built. 11 Over 60 
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years later, occupational competency measurement is still in its 
adolescence, 

4 

When one considers the time and the cost required to develop 
and validate occupational competency measures and the individualized 
testing and scoring typically required for performance measures, it 
is not surprising that the full utilization of competency tests has^ 
been slow. An'd yet, as Knaak (1977, p. 39) points out, the develop- 
ment and testing of criterion-referenced knowledge tests and perform- 
ance checklists is a task of "monumental importance in the competency- 
based learning system, 11 



Recent Efforts in Vocational .Competency Measurement 

This section reviews some of the more significant efforts cur- 
rently under way, or recently concluded, in the area of assessing 
occupational competencies of students in vocational training pro- 
grams. Only major projects with direct implications for improving 
student competency assessment in vocational education are covered 
here. In addition, commercial test publishers and other organiza- 
tions are currently working to develop proprietary tests, or testing 
programs,' either under contract to business and professional asso- 
ciations or for direct sale to the public. 

Three organizations are now engaged in large-scale programs for 
developing comprehensive measures of yocational competency, including 
both paper-and-pencil tests and performance measures: 

• The American Institutes^for Research 

• The National Occupational Competency Testing Institute 

• The Florida Department of Education 

A complete list of the tests in the final stages of development by 
the above organizations is contained in Module 18. The test develop- 
ment activities of the three organizations are described on the 
following pages. More detail will be presented on the American Insti- 
tutes for Research Vocational Competency Measures project because 
it was the project that provided the methodology underlying the four 
modules on student competency testing. An overview of project methods 
is contained in the Appendices. 



The American Institutes for Research--(AIR) 

In October 1979, AIR was awarded a contract from what is now the 
Office of Vocational and Adult Education of the U.S. Department of 
Education to develop, field test, and disseminate comprehensive 
measures of competency in selected occupational areas. 

The Vocational Competency Measures (VCM) project was designed 
to serve two major purposes: , (1) to hel-p teachers and administrators 




of secondary and postsecondary vocational education programs . 
evaluate and improve specific areas of their vocatidfi&i programs; 
and (2) provide an objective basis for informing students, 
teachers, and prospective employers about the progress made by 
students in acquiring specific; job-related competencies. 

The specific VCM project objectives are:- (1) to develop 
competency tests in 17 occupations representing the seven major 
curriculum 1 areas, (2) to establish their usefulness through exten- 
sive field testing and validation-, (3) to promote their acceptance 
and use in vocational education programs, and (4) to design and 
help implement a-prograWfor continuing test development on a 
self-supporting basis. 

Tests have been developed for the 17 occupations shown in 
Table 1. 

Each AIR test package includes a paper-and-pericil test of job 
knowledge; hands-on, performance measures; and a rating scale 
covering work habits and job-related "attitudes important to success 
on the job. 

A look at the AIR tests . A guiding factor in che development 
of the 17 tests was that each should assess skills and knowledge 
that employers and supervisors expect from newly hired employees 
who have completed a vocational training program. The paper-and-^ 
pencil tests use a multiple-choice format with most questions having 
five options. For all but two of the test packages, the paper-and- 
pencil tests have been divided into two parallel booklets that take 
about 45 minutes each to complete. The use of two booklets allows 
administrators and instructors a great deal of flexibility. We 
strongly recommend that each student be given both test parts, 
particularly if the test results will be used for student end-of- 
course certification or where decisions will be made concerning the 
proficiency of stqdents in individual areas covered by the test. 
However, it is possible to use one booklet as a screening measure 
for advance placement and to fiftd out what individual students know 
of the overall occupation. 

The performance tests included in each test package require 
the examinee to actually carry outcome of the entry-level activities 
expected on the job. These tests use only the equipment and mater^ 
ials nprmally available in a vocational or technical school offering 
the- specific program. All examinees are not necessarily expected to 
be tested on- the entire set. Deciding whi~h performance' tests to 
administer defends on the uses that will be made of the test scores 
and on actual job requirements covered. in the vocational program. 

The test packages contain from four to twelve individual perform- 
ance tests, some taking as little as five minutes to administer, and 
a few taking close to an hour. Instructors can select those tests 
that best apply to their particular training programs. Because these 
tests are modularized, instructors may administer them at appropriate 
times throughout a training program, rather than waiting until the end of 
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TABLE 1 



OCCUPATIONAL AREAS COVERED IN 
THE AIR VOCATIONAL COMPETENCY TESTS 



AGRICULTURE 

Agricultural Chemicals 
Applications Technician 

Farm Equipment Mechanic 



BUSINESS AND OFFICE 

Computer Operator 

Word Processing 
Specialist 

DISTRIBUTIVE EDUCATION 

Apparel Sales 

Fabric Sales 

Grocery Clerk 

Hotel (Motel) Front 
Office 



\ 



HOME ECONOMICS 

Custorg Sewing 

Restaurant Services 
(Waiter, Waitress, 
Cashier) \ 



TECHNICAL 



\ 



Electronics Technician 

Water Treatment 
Technician 

Wastewater Treatment 
Technician 



TRADE AND INDUSTRY 
Carpenter # . 
Diesel Mechanic 



• HEALTH 



Dental Assistant 

Physical Therapist 
Assistant 
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the program. Each performance^ test, includes instructions for the 
set-up and administration of .the test, scoring forms and proced- 
ures, instructions to the examinee, and a detailed list of required 
materials and ■ equipment . • 

The third component of each competency measures test package 
is a work habits '-inventory . This is an assessment instrument cover- 
ing some of the nontechnical competencies^ survival skills judged 
important for job, success in each occupation. Many of these behav- 
iors may be at least as important as technical skills* in determining 
success on the job. The work habits inventory can aid instructors 
in identifying additional training 1 needed to improve particular work 
habits and attitudes. 

Some of the behaviors included in the current work habits inven- 
tory are: 

• Organizes work to make best use of time 

• Shows cooperation and consideration in working with 

others 
■ 

• Gets work done 

• Expresses point of view effectively in a group 

• Maintains personal appearance suitable for the job 

This instrument is not designed to be used in the same way as 
our technical measures. Rather, it is intende£ to help improve the 
communication between student and teacher, and eventually raise- the 
competency of students in many of the job survival skills that may be 
neglected in the vocational curriculum. 

** 

For schools that would like to try out the work habits inventory, 
we suggest the following -steps : 

First, administer to new students in a. vocational program the 
list of behaviors our validation findings have shown to be related to 
that vocation. In this administration, the students would be asked 
how important they think each behavior wiil be to their future 
employers. Instruction on proper work habits can then be based on 
these results, and should ensure that students know what the employer 
expects. 

Later on' in the, school year, the students can rate themselves on 
these same behavioral statements, and at the same time, the instructor 
can rate each student, these individual student self-ratings can then 
be compared with the instructor ratings and together serve as a com- 
munication tool between student and teacher or student and counselor. 
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The National Occupational Competency Testing Institute (NOCTI) 

For the past 15 years, NOCTI has been actively involved in the 
development and administration of teacher competency tests in 
selected trade and industrial occupations. The current series of 
NOCTI teacher tests consists of 47 examinations covering 38 differ- 
ent occupations. More recently, NOCTI has begun coordinating the 
development of competency measures for the Student Occupational 
Competency Achievement Testing (SOCAT) program. In 1979, seven 
states pooied their resources to develop student tests. The states 
currently involved in this consortium are Alabama, Florida, Maryland, 
New Jersey, and Oklahoma. A total. of 21 student competency tests are 
now nearing completion. 

The Florida Department of 'Education * ' 

In addition to its responsibility for developing tests as part 
of the SOCAT Consortium, Florida has its owri statewide program for 
testing student occupational competencies as a means of improving 
the quality of vocational education programs. This program is man- 
dated under Florida statute for the purposes of educational account- 
ability and for identifying minimal competencies that students must 
have in order to perform effectively in the occupations for which 
they are trained. Ultimately , the goal is to provide state certifi- 
cation of competency achievement for sjtudents trained in Florida's 
vocational education programs. Plans call for test items and scoring 
to be provided by the State Department of Education, ^with test 
administration and materials furnished by local school systems, and 
monitoring conducted by local advisory committee members (Agee, 1980)-. 
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Individual Study Activities 



1. Select a reference from. the Recommended "References in the 
Appendices (or one of you* own choosing)" Read a chapter or 
section pertaining to an area in which you are especially 
interested, such as: military development and uses of com- 
petency measures; methods of empl^qnent testing; competency- 
bas,ed vocational education, etc.* Summarize your readings so 
that you can discuss your 6pinions and findings with the class. 

2. For your own state or district, select a vocational program 
or curriculum that you feel might benefit from an improved 
competency measurement effort. Outline the types of measures 
and the major topics that you feel should be included in a 
comprehensive competency measurement program. 



Discussion Questions 

1. Debate the advantages and the problems of evaluating vocational 
education programs by deciding on program success on the basis 
pf (a) number of graduates employed in fields for which they 
were trained versus (b) liow veil graduates can do the job for 
which they were trained. 

2. Discuss the rationale for including measures of work habits 
and attitudes in a. comprehensive vocational test package. Are 
work habits and attitudes vocational competencies? 



Group Activities 

1. Divide the class into three groups and assign each group one 
of the following topics: 

(a) Job Knowledge Testing 

(b) Performance Testing 

(c) Work Habits and Attitudes Testing 

The groups should be allowed time to discuss their topics and 
develop a list of categories they feel would be important to 
cover in developing a test for each assigned topic. It would 
simplify the activity if each group started, with a particular 
occupation in mind. 

N ]^fs N possible, the class instructor should obtain a sample test 
froiTNeach of the three groups now developing competency measures 
(AIR^OCTI, Florida)- Divide the class into thre'e groups, with 
each'grobp reviewing one of the three test samples. Each group 
should cri^ue the assigned test, recording their opinions on 
dimensions subh as clarity, comprehensiveness, appropriateness, 



GOAL 2- Identify the components of comprehensive student voca- 

tional competency measures, and define testing techniques 
. making up those components* 



- S peaking The. Laneuage Of r .n m pp.rencv Measurement 

Before going any further, let's review what instruments should 
be included in a comprehensive system of student competency assess- , 
lent. We also need to define some important terms. Even the term 
"vocational competency measurement" could stand some clarification. 
In these modules, Vocational m mperencv measurement refers to the 
pro ess of determining the extent to which a vocational student has 
acauired the knowledge and skills needed on the job. Normally, such 
measures would be administered near the end of a training program or 
at significant points along the way. We are not talking about admin- 
istering an aptitude test. to estimate how well a student can learn a 
particular course, but rather how much the student knows and can do 
. right now . £ 



The Total Picture 

What should be included in a comprehensive program -of student 
competency measurement? To really serve, the needs of vocational 
educates! we believe that three major types of instruments should 
be included: • t 

• Job Knowledge -Test . 

• Performance Test 

• Work Habits Inventory 

Let's touch briefly on each of these three measurement areas. 

1. .Tnh knowledge testing . This is reasonably straightforward and ' 
typically consists ot a paper-and-pencil test. If the content 
of the test is directly relevant to the job, a job knowledge 
test can be a very cost-effective measure and provide a great ■■ 

. deal of information per unit of time. You may have heard this 
type of test referred to as a "cognitive measure, which is fine 

. if you want to impress your friends. But the term "job knowledge . 
is perfectly'adequate for our purposes. 

A job knowledge test is rarely, if ever, a total measure of 'job 
competency* And, just because a test is a paper-and-pencil 
ZlTe Zes not/mean that it's a measure of only 
Paper-and-pencil items can also measure some of the skills, such 
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as problem solving, that are needed on most jobs today. In 
job areas such as business and office occupations, most of 
the required skills can be measured with paper-and-pencil tests* 

Performance testing . In a typical job-sample performance test, 
we expect the examinee to respond in ways identical or similar 
to those required on the afctual job. Despite the problems of 
time and cost, performance tests can be expected to provide 
valuable additional information not normally available from job 
knowledge tests. For many jobs, performance tests can more 
nearly match the actual job requirements withqut the test bias 
that can result when paper-and-pencil tests are used for jobs 
that actually have low verbal demands. 

The esoteric label "psychomotor test' 1 has occasionally been used 
in place of trie more general term "performance test." This term 
is unnecessarily limiting (not to mention the effect that it has 
on the "Fog Index") because psychomotor tests imply an emphasis 
on motor skills or eye-hand coordination. Obviously many jobs 
that exact little demand in the psychomotor area can still bene- 
fit from high quality performance tests. 

Performance tests can be further subdivided into: 

# Work samples 

• Simulations 

Work samples are generally defined as tests that use an actual 
job situation, with the same tools and materials to perform some 
of the same tasks as those required on the jOb>, Simulations 
require the examinees to roleplay or pretend that they are per- 
forming a real task. Simulation techniques, however, can range 
from highly realistic tests, wljich tend to overlap work samples, 
to situations where compromises are made in the stimulus or 
response characteristics in order to reduce testing costs or to 
gain more control over the testing situation. 

Work habits inventories . The combination of job knowledge tests 
and job performance tests still does not tap the full range of 
demands made upon workers. In order to obtaii^'a job and — even more 
important — to survive on that job, a person must possess a variety 
of nontechnical competencies that may be as valuable as technical 
skills. Coming to work on time, giving the "honest day f s work for 
an honest day f s pay," and working effectively with others are 
examples of' these nontechnical competencies fchafc cut across most 
jobs. Because this assessment area is so heterogeneous, we have 
labeled it simply- as "work habits" rather than use the elegant 
tenn "affective instrument." 

The work habits instrument developed by AIR as part of its 
Vocational Cqmpetency Measures (VCM) project was discussed earl- 
ier, and suggestions were made, for how such an instrument might * 
be used in a vocational program. Because of the exploratory 




level of this inventory and of the entire work habits meas- 
urement field, no' further discussion on the. topic is included 
in this . four-module series. 

Defining Criterion- and Norm-Referenced Testing 

The definitions of- criterion-referenced and norm^-ref erenced 
testing can also breed confusion. As most teachers know, criterion- 
referenced testing aims at explicit information concerning what an 
individual can -or cannot ' do ,. independent of the performance of others; 
norm-referenced testing provides information on the relative standing 
of individuals with respect to a given task. This definition may 
give the impression that criterion-referenced ..testing and norm- 
referenced testing are mutually exclusive, which need, hot be the case. 
For example,- the performance of examinees can be useful information 
for setting realistic performance 'standards in criterion-referenced 
tests. ... 

To give more precision to the definition Of criterion-referenced 
testing, a. number of educators have "subdivided' criterion-referenced 
tests into two major categories; ^/ 

/& 

# - Domain-referenced tests 

• Objectives-referenced tests 

In a domain-referenced test, the score indicates what percentage" of the 
specified domain the examinee has mastered. In contrast, the items 
in an objectives-referenced test, although matched to objectives, are 
not considered a representative sample of items from a clearly 
defined domain. 

Because this area is still in a state of flux, the best advice 
to educators is not to assume that all authors or. speakers mean the 
same thing when they use the terms "criterion-referenced," "domain- 
referenced," or "objectives-referenced." 



\ 
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Individual Study Activities 

1. Write your definitions of the following terms: 

(a) vocational competency measurement 

(b) performance t^/ting^ 

(c) criterion-referenced testing 

(d) norm-referenced testing 

You may want' to use a standard text on testing or references 
listed in the Appendices to help you arrive at a definition. 

Under Goal 2 in Module 9, you will find a discussion of ways 
to assess student achievement that might also be useful for 
. this activity. 

2. Select a vocational course or progranTwith which -you are 
familiar. List several competencies that are taught in the 

* program and should be tested. Then divide the list, into two 
sections: competencies best tested by papef-and-pencil and 
competencies best tested by performance. In making your 
judgements, be sure to consider the limits presented by situ- 
ations such as time restraints and the need for individual 
test administration. 



Discussion Questions 

1. Discuss the advantages and disadvantages of the following 
testing methods: 

(a) Work sample vs. simulation 

(b) Paper-and-pencil vs. performance testing 

(c) Criterion- vs. norm-referenced testing 

(d) Domain vs. objectives-referenced testing 

2. , Some'educators are hesitant when asked to use any affective 

instrument to assess attitudes'. . Discuss .how to use an 
affective instrument such as a work habits inventory to 
enhance student learning and vocational program improvement 
without raising fears about "values," etc. 



Group Activities 

\ 

1. Ask small groups (2-3) in the class to select a vocational 

program at a local high school, proprietary school, technical 

• institute, or community college where competency testing is 
used. Each group should be responsible for arranging to. 
observe a class (or individuals) in the selected program 
during a testing situation! Note and record your observa- 
tions, including how- the testing is set up and introduced 
to the student(-s), and the methods for rating which competen- 
cies 'are accomplished or not achieved. Then report during - 

; the next class session on each group's observations 7 




GOAL 3: Analyze the use of competency measures tor evaluation, 
instruction and administration, and communication. 



S pecific Application Of Vocational Com petency, Measurement 

■ i Vocational competency measures can assist "educators, and admin-; 
■istrators in three major areas: * 



• Evaluation a 



* Teaching and Administration , ' 

# Coiranunica tion ^ r* ' m 
These areas can be further subdivided into the "functions shown in . 

will necessarily be as valuable for other function s. The value of 
a test will depend both on its content and how it is usea. « f - 
ticSar the strategy and timing of test administration will ■ ^ 
influence the usefulness of a. test.. 

Ev aluation * 

When educational' evaluation is discussed, it is easy to jump 
to thf concision that the only decision, to he made are whethe , 

sjis Tu^t! p =; ^™r^sr-"%. 

financial support, limited view of evaluation. If 

SffllVESS c- etency tests can provide-much 

^information to help »ith ''^-^J-^rfc ur Ire' 
^^nrep nr Droeram need improvement and What pares 

achieving oSfctives o£ imparting occupational knowledge and skrlls 



to 'students. 



Teaching and Administration „ • 

Throughout the entire teaching .and administration Process 
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EVALUATION- 



TEACHING 
AND 

ADMINISTRATION 



COMMUNICATION 
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Figure 1. Role of competency measuremerit in voca- 
C tlonal education^ „ 
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objectives.' The major milestones where competency measurement is 
vital are: 

' • Initial assignment 



• -Remediation 

• Advancement 
Certification 



Initial assignment . Knowing the capabilities of your new 
students can be a big help— and, not just to teachers in an open- 
entry program. There's no sense in wasting your time or the stu- 
dents' time- teaching skills that they already possess. And trying . 
to teach an intermediate or advanced concept or skill if some of 
the students lack the enabling skills or prerequisites can also be 
wasteful, frustrating, and even dangerous. 

A judicious; administration of a selected group of "pretests" 
could really help you to' adapt or individualize your program, if 
only on a modest scale. You might also use the results to form more 
homogenous small groups, on the basis of how much of your- attention 
these groups need. That, way, you can concentrate your efforts on 
those students who really need your help while allowing the more 
advanced groups to work more on their own. 

x Remediation . Spotting gaps in student knowledge or skill 
areas is another way that competency measurement can support the 
teaching process. This is true whether you prefer working either 
with your total class or with individual students. Test results, , 
if sufficiently ."fine grained," can spot weaknesses objectively 
and efficiently. The scores are there for both you and the stu- 
dents to see specifically where improvement is needed. By using 
test scores, and other available information on student needs, 
you and your students can agree on, what actions should be taken 
•or alternative instruction methods used to correct deficiencies. 

Advancement . Progress measures, can be the same measures as 
those you might use for remediation. Not every test that's appro- 
priate for use in advancement, however, is adequate for diagnostic 
purposes. For diagnosing weaknesses and preparing remediation 
plans, we need to have a considerable amount of detail to pinpoint 
where a student is having trouble. . In contrast, an ^advancement _ 
test. may be more global and need only certify that a particular 
performance objective has or has not been achieved. 1 

- - * 

Certification . Schaefer and Huang have stressed the importance 
of measuring student proficiency at the end of a training program. 
They noted that "...we, the vocational educators, have steadfastly 
denied the majority of our graduates the chande to prove to them- 
selves thct 'I can do something and do It well"" (1978, p. 40). 
Such measures, they note, "would certainly help tVput our house 
in order, more so than any other tool of education" (p. 41). 
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Not every -vocational educator will be as enthusiastic about 
end-of-program measures as these educators; nevertheless, a growing 
number are looking toward certification as a meaningful indicator 
of accomplishment. 
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Communication 

While communication is an underlying component of all test 
uses, there are particular times when the communication functions 
of a' test are paramount. For example,, students need feedback 
periodically during a training program both for reassurance and to 
indicate where outside help or .greater effort is needed. Test 
scores can serve this purpose or, if necessary, can help provide 
the back-up data if a student requires counseling to change career 
plans. For minor students, the same information should be given 
to the parents. 

Finally, the results of objective, end-of-program competency 
measures can supply the detailed communication that employers are 
searching for as they decide on hiring of applicants and the place- 
ment of employees. What can a job applicant do arid what does the 
applicant know that will make that person a valuable employee? 
Grade reports or school transcripts can't begin to answer these 
questions as adequately. as can the results of competency tests. 

One caution should be noted. Be sure that you have the 
approval of the student (or the parent, if the student is a minor) 
before you release this information to a prospective employer. 



Don't Expect a Panacea 

Speaking about cautions, it's time to stress that, while 
tests can provide information that is not readily available from 
other sources, they are definitely not a panacea or a substitute 
for careful decision making. A recent report on ability testing 
from the National Research Council's Committee on Ability Testing 
addresses this issue directly (Wigdor & Garner, 1982): 

Americans 'have, on the whole, expected too much of tests 
and, conversely, have blamed too. much on tests. People ^ 
have wanted tests to produce social justice, to be fair* 
in some absolute sense. And,, when disappointed in the 
results of the testing process, people have charged them . 
with "being "unfair," with producing inequality (p. 206). 

When people stop thinking of tests .as panaceas or using 
them as scapegoats, when they understand that testing is 
*a useful, but limited, means of estimating one of the 
characteristics of interest in selecting or assessing 
people, i.e., ability or talent, then a good part of the 
.conflict about testing will be -alleviated (p. 208.). 
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While we have no doubt that atudent competency tests can make 
valuable contributions to decision making in vocational education, 
each time that test data are used, we should keep in , mind that we 
can't measure everything of importance and what we do measure 
must be based on a limited sampling of behavior. On many occa- 
sions, AIR staff have been approached by vocafioirai— educat-uis dtrd 
administrators wanting to use our test content when they design or 
revise a curriculum. If the test covered all of the competencies 
that are expected of graduates when they obtain their first job, 
this wouldn't necessarily present a problem, providing the test 
were used only for general guidance and not directly included in 
the- curriculum. ' Realistically, no test can cover all the competen- 
cies adequately, when-testing- tira£-±s^justi£iably xesltrlc.ted to a 
few hours. Test limitations due to restrictions on the length of 
the test must always be kept in mind by curriculum developers. 

On the topic of test limitations, the National Research 
Council's Committee on Ability Testing again makes a cogent obser- 
vation: 

The limitations of testing technology and the problems 
caused by its misuse lead us to a cautionary conclusion. 
Tests are tools.' They provide- an efficient way to 
gather certain kinds of information systematically and 
theyv extend to decisionmaker one means of making judgments 
about people. But when a test score is taken -out of con- 
text and treated as if it tells all that matters about a 
person, scientific assessment is degreed to dogma. 

Recognizing these limitations on thfe use of test scores, we * 
should be sensitive to the legal issues^ surrounding competency test- 
ing in vocational education. In look^g to the future, Tractenberg 
makes the following recommendations: , _ 

Vocational educators should not simply sit back and wait 
to be' sued. They should 'deal in some preventive mainte- 
nance—they should attempt" tQ head off legal challenges 
by fashioning and implementing performance testing programs 
in the most careful manner possible. If they do so, the 
law and the courts will have been an important partner in 
educational and professional reform (p. 103). 

Now that you've seen the many ways where tests can be helpful, 
as well as some of the cautions that/ should be kept in mind, we hope 
you will still want to pursue the topic further. The next module in 
this series takes you through the first step in wise test usage, 
determining requirements for vocational competency measures. 
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Individual Study Activities 

1, Select a basic reference on evaluation of vocational educa- 
tion. Some are listed in the Appendices of this module. and 
in Mod ul&S 9 and _ i2. Kead the specific sections dealing 



either with evaluating students or with evaluating curricula. 
Prepare a brief paper on the evaluation methods, limitations, 
and uses presented in your Selected reference. 

2. Find several newspaper or journal articles on .the controver- 
sies surrounding minimum competency testing of basic skills and 
prepare a brief paper or^ t he im plications^ f orjvoca tionaT- compe- 
tency Testing". 



Discussion Questions 

1. Discuss the problems inherent in communicating test results to: 

(a) students 
" (b) parents (if applicable) 
(c) employers 

For each problem, arrive at a communication method that would 
alleviate or prevent the problem presented. 

2 Discuss some of the limitations to using the content of a 

competency test to design or revise a vocational curriculum. 

3. Compare the benefits (and drawbacks) of using competency test 
scores rather than course grades or school transcripts for 
communicating with prospective employers about vocational 
graduates. 

Group Activity 

1. Assign each of three groups one of the following functions of 
competency testing: 

• Evaluation 

• Teaching and Administration 

• Communication 

Ask each group to develop a checklist that can be used in t;heir 
own school, district, or state to examine an existing voca- 
tional testing program on the assigned function. For example, 
the group developing a checklist for using competency test 
" results for evaluation might consider the following questions: 
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How and when can evaluation methods be used to measure 
program (or course) effectiveness? 

How can evaluation results be used to help decide what 
i nstructional methods, program content; etc. , s hould be 
changed? 
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Summary 



The concept of measuring .achievement in vocational training 

pr og r ams ' U u uug h com pete ncy t e st i n g io not n ew^-His torically , rhft 

military services took the lead in developing and using competency 
measures for assessing student progress. Most recently, the trend 
toward more competency-based vocational education programs has 
increased the demand for assessing students on the basis of 
competencies. 

Sojng of the current efforts in vocational competency measures 

are through projects conducted by the American Institutes tor 
Research (AIR) , the National (Occupational Competency Te.stmg 
Institute (NOCTI), and the Florida Department of Education. .Test- 
ing components in the AIR Vocational Competency Measures project 
include: a paper-and-pencil test of job knowledge; a set of . 
hands-on performance tests; and a rating scale or inventory o£ 
work- habits and attitudes. 

The uses of a comprehensive package of student competency 
measures are varied. Three basic functions are evaluation, teach- 
ing and administration, , and communication. Within these Actions 
are a range of activities from using results of competency measurement 
' to improvf a vocational curriculum, to helping educators communicate 
with -students about their progress. 
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Summary of VGM Proj ect Methods 

The Vocational Competency Measures Project has been divided 

" iffto-three major staged ^xnrpirtionat-study , (2) test develop 

' " ment and analyses, and (3) dissemination and technical assistance. 

Occupational study . In order to ensure that our tests reflect 
actual job requirements, task .inventories were prepared- for each 
occupational area selected for test development. Previous task 
analysis data were used wherever possible, and the on-the-job per- 
formance of these tasks was verified through interviews with 
employers and employees across the country. Altogether, interviews 
were conducted in 26 states and the District of Columbia. The task 
verification data were analyzed to identify tasks that should be 
included in the competency measures. 

Te st development and analyses . The test development and 
analysis stage involved technical' experts working with AIR staff 
to prepare the final test outline, the individual test items, and 
the job-sample problems. Each test was then reviewed by at least 
four experts representing employers and educators who had not 
developed the test items. This review was important to guarantee 
that the terts reflect real job demands and, at the same time, are 
sensitive to the problems faced by vocational educators. Based on 
the reviews, each test was revised and pilot tested on small groups 
of students in at least two separate sites. 

The competency measures were then field tested in vocational 
education programs across the U.S. Altogether, over 3500 students 
at more than 150 sites in 37 states, participated in the field test- 
ing phase. The results of the field testing provided the basis for a 
thorough revision of test content and format. 

The next step was to validate the tests in the job environment 
so that vocational educators could depend on the tests as meeting 
current job requirements. Two strategies were used for test valida- 
tion. For content validation, the primary approach, employers and 
supervisors critically reviewed the tests to, determine how well the 
, content matched the actual job demands. As a supplement to the 
content validity review, employers" volunteered to administer portions 
of the tests to samples of employees to determine whether performance 
on the test correlated with performance in the work environmerit-m 
other words, whether those who do well on the test also do well on 
the job. This strategy is often referred to as concurrent validity. 
Over 1600 content validity checklists were completed by .employers 
throughout the 50 states, and approximately 150- civilian employers 
and military units administered the tests and returned sufficiently 
complete data for analysis. 

Dissemination and technical assistance . This' third stage has 
actually been under way since the early months of the project. 
Virtually every project task has had a related dissemination activity. 
Our specific dissemination products and efforts have included _the 
. • following: 
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• Project abstract 



• -Project brochure 



« — l^DAT-I^-^ui^eri-odie— ^ with 

over 2000 individuals and institutions) 

• Journal articles 

• News releases 

• Presentations at state and national vocational 
education conventions 

• Targeted presentations to state and local 
vocational education departments 



Technical assistance manuals covering the effective 
use, development, and evaluation of vocational 
competency measures 
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Self-Check 

■GOAL 1 

"T; DeBtrlbe-Chr advantages that competency testing has over employment 

rate as a criterion for evaluating vocational education effectiveness, 

2. List at least three ways that competency-based vocational * 
education is linked to competency testing. 

3. Name four organizations that have been or are involved in 
developing. competency measures for vocational training programs. 

4. Briefly describe the three components of AIR's comprehensive 
Vocational Competency Measurement program. 



GOAL 2 

1. Define vocational competency measurement in your own words. 

2. Describe briefly how the three major types of instruments that 
should be included in a comprehensive set of vocational compe- 
tency measures differ f-rom each other. 

3. • For each of the above instruments, state one advantage and one 

problem in its use with vocational students. 

4. For your own understanding of the terms, define criterion-^ 
referenced and norm-referenced testing. 

GOAL 3- ■ ^ 



1. For each of the functions of vocational competency measurement^ 
listed below, suggest one use you might make of a vocational 
competency measurement program: 

• Evaluation * 

• Teaching and Administration 

• Communication 

2 Can the same competency measures be used for initial placement, 
diagnosis, advancement, and certification? In what ways should 
each of the tests or their uses differ for each purpose. 

3. What are the major, limitations of student - competency tests? 
What are some major contributions of vocational' student 
competency tests? 
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Self-Check Responses 



GOAL 1 



1 Not- all students in vocational- programs are serious about makxng 
a career in the field covered by the program.. Local economic ^ 
conditions influence the likelihood of a student (especially ■ 
a high school student) getting .a job in the field where trained. 



Pre-testing entering students 

I 

Advancing students on achievement of objectives . , 

Providing remedial or alternate instruction for students 

Measuring proficiency at the end of a course or program 

Recording or certifying student achievement of objectives 
or competencies 



3. U.S. Military services; AIR;* NOCTI; Florida State Department 
of 'Education 

4. Job knowledge test (paper-and-penci*l) ; cognitive 

■ Performance test (simulation or work sample); hands-on; 
psychomotor 



Work habits and. attitudes (inventory of behaviors); affective 



GOAL 2 

1. Vocational competency measurement is the process for determining 
the extent to which a vocational student has acquired the knowledge 

and' skills needed on the job 

» * 

2. Job knowledge test usually measures factual information about a . 
job; paper-and-pencil test 

Performance test usually measures actual on-the-job skills by a 
work sample or simulation technique 

Work habits inventory is directed toward evaluating work-related 
behaviors that are nontechnical • 

3. Paper-and-pencil test: limited mostly to knowledge, not skills; 
dependent on verbal ability of student; cost-effective, easy to 
administer and score 
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Performance test: limited by cost and time required for , 
■ ' administration; does not usually cover all skills required 
on a job; measures student .skills without depending on 
verbal ability; best predictor of performance on the job - 

4 Criterion-referenced i measures explicitly what an individual 
can : or cannot do toward meeting a, performance objective 

Norm-referenced: measures performance of individuals based 
'on their relative standing among all others tested on the 
same measures 
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GOAL 3 ' .-, ' 

1. Evaluation— using results to review particular curriculum, 
find weak areas, revise courses, etc. 

T 0a rM. nf > and administration— deciding where to place entering 
students; how and where to provide remedial coursework or 
alternative instruction; certifying students as objectives are 
achieved »>• ■ 

Com munication- infb^rming'students about strengths* and weaknesses; 
providing rationale for parents about need for improvement; 
referring students to potential employers 

2. Remediation: test must focus on details 

Advancement/certification: outcomes (end-of -program) results ar,e 
important 

3. Limitations of vocational competency tests: 

- performance tests require more skill, time,' individualized 
attention, preparation than paper-and-pervcil tests ^ 

- measuring competencies adequately requires combination of 
tests and methods to judge levels of knowledge, skills, 
attitudes; everything can't be measured ^ 

Contributions of vocational competency tests: * 

- bias-free, testing " 

- measurement of job-related (not education-related) perfprm- 
ance 

- clear definition of strengths and- weaknesses to help in . 
deciding about programs, curricula, -student progress 
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